Method of controlling a control point position on a command area and method for control of a device

ABSTRACT

The invention describes a method of controlling a position (x′, y′) of a control point (c) on a command area (A CM ). This method comprises the steps of:
         aiming a pointing device ( 1 ) comprising a camera ( 2 ) in the direction of the command area (A CM );   generating an image (I) of a target area (A I ) aimed at by the pointing device;   processing the target area image (I) to determine a target point (T) at which the pointing device ( 1 ) is aimed;   determining position (x′, y′) of the control point (C) according to the position (x, y) of the target point (T) within a currently defined control area (A CT ), which control area (A cr ) includes at least partially the command area (A CM ).       

     Moreover the invention describes an adequate system and a pointing device ( 1 ) for controlling a control point position (P) on a command area (A CM ).

This application claims the benefit or priority of and describesrelationships between the following applications: wherein thisapplication is a continuation of U.S. patent application Ser. No.11/995,010, filed Jan. 8, 2008, which is the National Stage ofInternational Application No. PCT/IB2006/052232, filed Jul. 3, 2006,which claims the priority of foreign application 05106316.2 filed Jul.11, 2005, all of which are incorporated herein in whole by reference.

This invention relates to a method of controlling a position of acontrol point on a command area, and to a method for control of adevice. Furthermore, the invention relates to an appropriate system anda pointing device for controlling a position of a control point on acommand area, and to an appropriate system for controlling a device.

For control of a device, such as any consumer electronics device, e.g.television, DVD player, tuner, etc., a remote control is generally used.However, in the average household, multiple remote controls can berequired, often one for each consumer electronics device. Even for aperson well acquainted with the consumer electronics devices he owns, itis a challenge to remember what each button on each remote control isactually for. Furthermore, the on-screen menu-driven navigationavailable for some consumer electronics devices is often less thanintuitive, particularly for users that might not possess an in-depthknowledge of the options available for the device. The result is thatthe user must continually examine the menu presented on the screen tolocate the option he is looking for, and then look down at the remotecontrol to search for the appropriate button. Quite often, the buttonsare given non-intuitive names or abbreviations. Additionally, a buttonon the remote control might also perform a further function, which mustfirst be accessed by pressing a mode button.

Sometimes, a single remote control device can be used to control anumber of devices. Since the remote control is not itself capable ofdistinguishing one device from another, the remote control must beequipped with a dedicated button for each device, and the user mustexplicitly press the appropriate button before interacting with thedesired device.

A possible alternative to the usual remote control may be a pointingdevice. A user interface system based on a pointing device is known fromWO 2004/047011 A2 which disclosure is herewith included by reference.The concept of such a system is that a pointing device connected to acamera can be used to control any device in its surroundings by beingaimed at an object, e.g. a housing of the device, a screen, or any otherapplication or accessory associated with the device. The camera of thepointing device generates images of the target area at which it is beingaimed, and these images are subsequently analysed.

In many applications for controlling a device, a number of user optionsfor the device to be controlled are visually presented on a command areaor graphical user interface. If the device—e.g. a television, personalcomputer or laptop—avails of a screen, a beamer or other image renderingmeans, such a command area can be presented with the aid of the imagerendering means. Furthermore, a command area for a device to becontrolled can be presented in any other way, upon which the options forthis device can then be presented. For example, a central control unitmight feature a display upon which the command area is presented.Thereby, the exact position of the pointing axis (i.e., the pointvisible in the centre of the captured images) can be determined in thecommand area. This allows for instance control of a control point, forexample a cursor, on the command area and thus navigation through menusand other styles of interaction which are usually possible with agraphical user interface, such as create sketches etc. Other importantpossible interaction modes may comprise “beyond-screen” interaction,such as scrolling up or down when pointing above or below the screen,performing drag-and-drop operations between screens, and gesturing byperforming pre-defined or user-defined geometric motion patterns “in theair”.

For allowing such “mouse-like” control of a control point on a commandarea from at a distance, it is essential to allow accurate interactionup to the maximally possible distance given by the user's opticalcapabilities, i.e. the resolution of the eyes in relation to thedimensions of the controlled object. With the help of visual aids, thistypically goes beyond the range for which accurate hand coordination ispossible. In fact, when the distance to the command area becomesrelatively large, the angular range in which the direct interactiontakes place, i.e. the angle under which the command area is seen fromthe user's position, gets relatively small. As a result, the interactionbecomes more difficult to perform accurately, owing to the strongnegative influence of jitter and unconscious movements of the handholding the pointing device.

Therefore, an object of the present invention is to provide an easy andreasonable way which allows the exact controlling of a control point ona command area, also from larger distances.

To this end, the present invention provides a method of controlling aposition of a control point on a command area, wherein—as mentionedabove—a pointing device comprising a camera is aimed in the direction ofthe command area, an image of a target area aimed at by the pointingdevice is generated and the target area image is processed to determinethe target point at which the pointing device is aimed. Thereby,according to the invention, the control point position in the commandarea is determined depending on the position of the target point withina currently defined control area, which control area at least partiallyincludes the command area. Therefore, according to the invention,whenever the user aims the pointing device at a real target point (alsocalled “actual target point” in the following) within the “virtual”control area, the real pointing position in relation to the virtualcontrol area will be converted to a virtual pointing position inrelation to the actual command area, whereby this virtual pointingposition corresponds to the control point position.

By ascertaining the real target point within the virtual control area inthe manner according to the invention, and by mapping this position ofthe target point to the current command area, it is possible to adaptthe control area, required for locating the target point, to the currentsituation. Particularly advantageously, the control area can be enlargedin relation to the command area, especially in the case when the user ispositioned far away from the device. Use of a virtual control areaadapted to the distance between the pointing device and the command areaensures that slight movements of the pointing device are not interpretedas correspondingly great movements in the command area, enabling theuser to direct the pointing device more precisely within the commandarea. This applies in particular to angular movements of the pointingdevice which, with increasing distance from the command area, result inproportionally large linear motions of the real target point in thecommand area.

A “control point” is intended to mean not just the usual cursor, but anykind of visual feedback for showing the user at which point in thecommand area he is currently aiming It could, for example, be a lightpoint projected onto the command area by, if necessary, a separatedevice. Equally, certain areas symbolising different menu items might behighlighted, as is often the case in menu-driven graphical userinterfaces, when the control point lands in such an area.

Since the control point is shown by means of a suitable visual feedback,the user of the method according to the invention will not, even at alarge distance, be given the impression that he is pointing outside ofthe command area, even so this is the case. He would only notice thediscrepancy if he were to look along the length of the pointing device.Instead, he is simply given the impression of better controllability ofthe control point in the command area.

As mentioned above, this method is advantageous particularly when theuser wishes to interact with the command area over a large distance.Therefore, enlarging the control area in relation to the command area isthe preferred application of the invention. However, it is equallypossible to create a virtual control area that is smaller than theactual command area. An example might be controlling a presentationsetup, in which the command area is displayed greatly magnified on abackdrop for a presentation, and where the person giving thepresentation is standing relatively close to the backdrop. In such asituation, it might be opportune to designate a smaller area for thecontrol area, so that the user can target all regions in the commandarea with the control point, but with smaller movements of the pointingdevice.

An appropriate system for controlling a position of a control point on acommand area comprises a pointing device with a camera for generatingimages of a target area in the direction of pointing and an imageprocessing unit for processing the target area images to determine thetarget point at which the pointing device is aimed. According to theinvention, the system must further comprise a control point positioncalculation unit for determining the control point position depending onthe position of the target point within a currently defined controlarea, which control area at least partially includes the command area.Furthermore, the system should comprise a control point visualizationunit for visualization of the control point on the command area.

The dependent claims and the subsequent description discloseparticularly advantageous embodiments and features of the invention.

In a preferred embodiment of the invention, the dimensions and/orlocation of the control area are defined according to the momentarypointing conditions and/or the control situation. The term “pointingconditions” means not only the distance, but also the angle between thepointing device and the command area. Depending on the position of thepointing device relative to the command area, not only the dimensions ofthe control area can be determined, but also the aspect, i.e. whetherthe control area is offset above, below, or laterally relative to thecommand area. The term “control situation” implies, amongst others, thetype of menu, the size of the individual menu items etc., in the commandarea. For example, in a situation in which a menu with a large number ofmenu items is being displayed in the command area, where each menu itemoffers only a relatively small activation region, i.e. the area in whichthe pointing device must be aimed in order to select the menu item, itmakes sense to offer an enlarged virtual control area, even in the caseof relatively small distances from the command area, in order to make iteasier for the user to precisely aim at an individual menu item or itsactivation region. On the other hand, if only a relatively small numberof options, with relatively large activation regions are being shown inthe command area, it suffices that the control area remains the samesize as the command area, even for larger distances.

As mentioned above, the dimensions of the control area are preferablydefined first of all according to the distance between the pointingdevice and the command area. A number of possibilities exist for themanner in which the dimensions of the control area are enlargedaccording to the distance between the pointing device and the commandarea.

In one embodiment, the dimensions of the control area are extended orenlarged if the distance between the pointing device and the commandarea exceeds a certain limit. According to this method, for example, thecontrol area can coincide with the command area for a first, relativelysmall, distance. Should the user move with the pointing device beyond acertain distance threshold away from the command area, the virtualcontrol area will be enlarged by a certain amount. Beyond anotherfurther distance threshold, the virtual control area can again beenlarged by a certain amount. An incremental step-wise enlargement ofthe control area, depending on the distance, is thus made possible. Thedistance threshold can thereby basically be chosen to be as small asdesired, so that an apparently continuous transition is possible. Thevalues for the distance thresholds and the corresponding dimensions ofthe control area can be stored in a table.

Alternatively, an appropriate continuous relationship between thedimensions of the control area and the distance from the command areacan be defined in a suitable function. For example, a distance-dependentenlargement factor can be chosen, which specifies the extent to whichthe dimensions of the virtual control area are magnified in comparisonto the actual size of the command area.

Preferably, at least for very great distances, the extension factorwhich defines the enlargement or re-sizing of the dimensions of thecontrol area according to the dimensions of the command area should beproportional to the distance between the pointing device and the commandarea.

In a further preferred embodiment—if the user is aiming at a targetpoint outside of the current control area—the dimensions of the controlarea are enlarged such that the target point lies inside the new controlarea.

One potential choice using this feature would be to always enable directinteraction, i.e. the virtual control area corresponds to the physicalcommand area for all distances so long as the user is aiming thepointing device at a point within the command area. Only when the userpoints outside the command area, another value for the enlargementfactor will be applied, thus allowing a relative scaling. This method,however, has the disadvantage of a discontinuity at the displayboundaries.

In a preferred alternative—as in the alternative above—the interactionis initially direct. If the user aims the pointing device at a pointoutside of the physical screen, the extension or enlargement factor isincreased such that the actual axis of pointing, and thereforeultimately the actual target point, coincides with a point on theboundary of the (updated) virtual control area. In this alternative, theextension factor is therefore not just a function of the distance alone,but depends also on user behaviour. Such an operation may be called a“punch” since the virtual control area behaves as if its borders werepushed or punched outwards when the pointing position (apparently) triesto traverse them. Preferably, a minimum distance and/or a minimum timemight apply before this condition triggers.

In order to allow “beyond-screen” interaction, e.g. scrolling up/down bypointing above/below the screen, this feature might be active only inone direction (in this case the X axis), or it might be that just one ofthe four sides of the rectangular control area is “punchable”. In anycase, the fact, that the user puts away the pointing device or startspointing at something else in the room, can be detected, so that theseevents will not lead to punches of the virtual screen area.

In a preferred embodiment, the dimensions of the control area aredecreased again after a certain time period during which the user is notaiming at a target point outside of a certain region of the currentcontrol area. For example, if the user does not punch against theboundaries of the momentary control area, the extension factor decreasesslowly with time, ultimately converging towards a value of 1. Preferablythe time constant is relatively long, e.g. in the order of seconds tominutes. The virtual control area will thus adapt to the user'spreferred dimensions. This alternative will also guide users,inexperienced in this direct method of pointing, but who may haveexperience with indirect methods of control, such as with a computermouse, to automatically make the mental transition from the relativepointing paradigm to the direct manipulation paradigm.

Since the pointing device allows controlling of very different devices,in particular devices which might be in the immediate vicinity of eachother, a conflict of control areas of neighbouring devices should beavoided. An overlap of control areas of two different devices couldresult in an ambiguous conclusion as to which device the user isactually aiming at. There are a number of possibilities for dealing withsuch a situation:

In a first—very simple—solution, the control areas of adjacent commandareas are chosen so that the control areas do not overlap. That meansthe virtual control areas are kept completely separate.

In a further preferred embodiment, only a certain region of a commandarea, for example the centre or central region of the command area, ofone device is kept out of the virtual control area of the other adjacentdevice and vice versa. Then, preferably a command area can be activatedby aiming at that certain region of the corresponding command area andthe extended control area associated with the currently active commandarea is chosen so that it does not overlap with the certain region ofthe adjacent command area. That means, in the latter case, the pointingambiguity can be resolved by always choosing the device that waspreviously targeted, i.e. only switching to the other device when thepointing direction uniquely determines the target device. The user willthen experience a kind of hysteresis effect: the control tends to stickto one device, and only by pointing to a certain “activating region”,for example the central parts of the command area, can the focus beshifted to the other device.

In a method for control of a device according to the invention, thepointing device with the camera is aimed in the direction of the commandarea associated with the device to be controlled, within which commandarea is presented a number of options for controlling the device. Visualfeedback of the position of the momentary control point is given in thiscommand area, for example in the form of a cursor. Subsequently—asexplained above—an image of a target area aimed at by the pointingdevice is generated, and the target area image is processed to determinethe target point at which the pointing device is aimed. According to theinvention, the chosen option is determined depending on the position ofthe control point within the command area, wherein the control pointposition is controlled according to the method explained above.

An appropriate system for controlling a device comprises a command areafor visually presenting user options for the device to be controlled, asystem for controlling a control point positioned on the command area asexplained above, and an interpreting unit for interpreting ordetermining the chosen options according to the position of a controlpoint within the command area. This interpreting unit may also generatea control signal for the device to be controlled according to the chosenoption.

Identification of the chosen option on the basis of the current controlpoint position, and generation of the control signal can be effected inthe usual manner, e.g. as is the case for a mouse controlledapplication, known to a person skilled in the art.

To determine the momentary actual target point in the control area,computer vision algorithms can be applied. A method of processing theimage data of the target area image using computer vision algorithmsmight comprise detecting distinctive points in the target image data,determining corresponding points in a template of the command area, e.g.the screen, of the device or in the (known) surroundings of the commandarea, and developing a transformation for mapping the points in theimage data to the corresponding points in the template. Thistransformation can then be used to determine the position and aspect ofthe pointing device relative to the virtual control area including thecommand area so that the point of intersection of the pointing axis withthe control area can be located in the template. The position of thisintersection in the template corresponds to the target point in thecontrol area which can be used to easily determine the correspondingcontrol point in the command area, for example to identify which optionhas been targeted by the user. Comparing the image data with thepre-defined template may thereby be restricted to identifying andcomparing only salient points such as distinctive corner points. Theterm “comparing”, as applicable in this invention, is to be understoodin a broad sense, i.e. by only comparing sufficient features in order toquickly identify the option at which the user is aiming.

In a particularly preferred embodiment of the invention, the processingof the target area images to identify the target point in the controlarea or the control point in the command area might be carried outwithin the pointing device itself. However, the computing power of sucha pointing device, which is preferably realized to be held comfortablyin the hand, is necessarily limited by the power required by complicatedcomputing processes. Therefore, the target images are preferablytransmitted for further processing to a control unit associated with thedevice to be controlled. The control unit can identify the target pointin the control area and the control point in the command area and mayfurther send a control signal to the device which generates the commandarea in order to give a visual feedback of the control point position tothe user, e.g. in form of a cursor position. This control unit may alsodetermine a control signal for the device to be controlled on the basisof an identified option selected by the user.

Such a control unit may be, for example, part of the device and may beincorporated in the same casing as the device itself. In anotherembodiment, this control unit may be realised as a separate entity whichcan communicate in any appropriate manner with the device to becontrolled, whereby such a control unit may be capable of controllingmore than one device.

The pointing device and control units described in the above combine togive a powerful control system, for use in practically any kind ofenvironment, such as a home, office, museum, hospital or hotelenvironment. The method according to the invention can be applied to anyelectrically or electronically controllable device. Furthermore, thecontrol unit and the device to be controlled can comprise any number ofmodules, components or units, and can be distributed in any manner.

Other objects and features of the present invention will become apparentfrom the following detailed descriptions considered in conjunction withthe accompanying drawings. It is to be understood, however, that thedrawings are designed solely for the purposes of illustration and not asa definition of the limits of the invention.

FIG. 1 is a schematic perspective representation of the principle of anextended virtual control area associated with a command area;

FIG. 2 is a schematic representation of a pointing device for use withthe invention;

FIG. 3 is a rough schematic representation of a device controllableaccording to the invention;

FIG. 4 is a schematic representation of two neighbouring command areasand their corresponding control areas according to a first embodiment;

FIG. 5 is a schematic representation of two neighbouring command areasand their corresponding control areas according to a second embodiment;

FIG. 6 is a schematic diagram showing a command area, realised by ascreen, its associated control area, its associated template, and atarget area image generated by a pointing device in accordance with anembodiment of the present invention.

In the drawings, like numbers refer to like objects throughout. Thepointing device described is held and operated by a user, not shown inthe drawings.

The underlying principle of the invention, as already explained above,is that a certain command area, in which are displayed various optionsfor control of the device, is converted to a virtual control area.Thereby, the position of the real or actual target point within thisvirtual control area will be converted into the coordinates of a controlpoint, for example the position of the cursor, within the command area.Generally it is a virtual enlargement of the command area that will beeffected, thus making it easier for the user to direct a cursor orsimilar point within the confines of the command area. Therefore, in thefollowing, the example of an enlarged control area being assigned to acommand area is used for the purposes of illustration, but withoutconfining the invention to this example.

As shown in FIG. 1, the principle of the transformation from a realtarget point T in a virtual extended control area A_(CT) to the controlpoint C in the real command area A_(CM) is to map the actual pointingray s to a virtual pointing ray s′ which is closer to the centre of thecommand area A_(CM). In general, this is achieved by defining a mappingfrom the spherical coordinates (θ, φ) of the actual pointing axis s tothe spherical coordinates (θ′, φ′) of the virtual pointing axis s′. Itis very natural to assume that the mapping should be rotationallysymmetric, i.e. depending only on θ. Therefore, one can set φ′=φ andθ′=f_(r)(θ) with some function family f being parameterized by thedistance r between the user, i.e. the pointing device 1, and the commandarea A_(CM).

In order to avoid trigonometric complications it is advantageous—and atthe same time very natural—to chose a dependence of the form

θ′=arctan(α(r)·tan(θ))

because this translates—under the assumption that the origin of thecoordinate system lies in the centre of the command area—to a lineartransformation when changing over to a two-dimensional coordinate systemin the plane of the command area A_(CM). The following then applies:

x′=α(r)·x

y′=α(r)·y

Thereby, the distance dependent factor α(r) can be regarded as a scalefactor. The value of α(r) must lie between 0.0 and 1.0; the smaller thevalue, the larger is the enhancement or enlargement factor by which thedimensions of the virtual control area A_(CT) are magnified or enlargedin relation to the real size of the command area A_(CM).

For an explicit choice of the dependency of α on r, the followingconstraints should preferably be taken into account:

For very large distances, the angle at which the virtual control areaappears, as seen from the user's point of view, should remain constant,since, on the one hand, it would be counter-intuitive if the angleshould increase with distance, whereas on the other hand, it should notdecrease below a certain critical limit. Therefore, the scale factorshould preferably be chosen so that:

α(r)˜1/r for r→∞

For small distances, the benefit of extension of the control area A_(CT)at some point is outweighed by the intuitiveness of direct interaction(“what you point at is what you control”). Therefore, for smalldistances, the scale factor should preferably approach 1:

α(r)=1 for small r

Probably the best way to bring these two constraints together is by asmooth, incremental combination in the form:

α(r)=min(1,r ₀ /r)

The distance parameter r₀ can be interpreted as the distance at whichthe behaviour smoothly switches from direct interaction on the commandarea A_(CM) to interaction on the virtual extended control area A_(CT).This distance parameter r₀ might be set, for example, depending on thedevice or on the current application.

Alternatively, or additionally, the distance parameter r₀ may also bechosen depending on the user's preferences and/or the user'sexperience/abilities. For example, the distance parameter r₀ can beadjusted by the user according to an expected degree of jitter, and/orthe user's pointing accuracy might be measured (in a calibration step)and r₀ adjusted accordingly.

FIG. 2 shows a greatly simplified embodiment of a pointing device 1 foruse in a method according to the invention. This pointing device 1, witha camera 2 in its forward end, features a housing in the shape of a wandor pen in an elongated form that can be grasped comfortably and easilycarried around by the user. Equally, the pointing device might be shapedin the form of a pistol.

Target area images I generated by the camera 2 are transmitted to animage processing unit 3, which processes the target area images I todetermine the position of the actual or real target point T at which theuser is aiming the pointing device 1. Ascertaining the target point Tand the distance r can be achieved with the aid of templates of thecommand area A_(CM) or the area surrounding the command area A_(CM), bycomparing the target area image I to these templates. The actual methodof determining the target point T will be explained in more detail belowwith the aid of FIG. 6.

The target point T is then forwarded to a control point positioncalculation unit 4, which determines the control point position, i.e.the coordinates x′, y′ of the control point C within the command areaA_(CM), according to the coordinates x, y of the target point T withinthe virtual control area A_(CT). The dimensions and position of thecurrent control area A_(CT) can also be determined in this control pointprocessing unit 4. The necessary data for this calculation, for examplethe distance between the pointing device 1 and the command area A_(CM)can also be deduced from the target area image I, for example from theimage processing unit 3. The image processing unit 3 and the controlpoint processing unit 4 can, for example, be realised in the form ofsoftware modules on a programmable processor of the pointing device 1.

The coordinates x′, y′ of the control point C can subsequently betransmitted by a communication interface 5 to a control unit 6 of adevice D₁ to be controlled (cf. FIG. 3). The communication interface 5,which might be a standard interface such as Bluetooth, IEEE or similar,might also be implemented to receive or transmit other data for thepointing device 1. For example, the templates for the various devicesand/or corresponding command areas A_(CM) with their surroundings mightbe supplied to the pointing device 1 using such an interface 5, so thatthe image processing steps can be carried out in the pointing device 1.Insofar as the pointing device 1 avails of other components such asbuttons, knobs or other type of user interface, by means of whichadditional signals can be generated for use in the control of the deviceD₁, these can of course also be transmitted via the communicationinterface 5.

It goes without saying that the pointing device 1 can comprise othercomponents besides those mentioned above, for example buttons, or a typeof user interface by means of which a user can input additional commandsfor the generation of a control signal for a device. In regard to suchadditional components, reference is made once more to WO2004/047011A2,in which a number of additional components for such a pointing deviceare described in detail. The pointing device can also be used for thegeneration of control signals in the manner described therein. Forexample, besides simply pointing at a target, the pointing device can bemoved in a particular manner or gesture which can be interpreted in aparticular manner for the generation of control signals.

FIG. 3 shows, in a very simplified schematic overview, the componentsnecessary for elucidation of the invention, of a device D₁ to becontrolled. Here, the device D₁ to be controlled comprises a monitor 10,upon which a command area A_(CM) is displayed with a menu featuring anumber of menu items M1, M2, M3, M4, M5, M6 for selection by the user. Afurther component of this device D₁ is a control unit 6, shownseparately from the monitor 10, as might be the case for a personalcomputer. This control unit 6 can of course be incorporated with themonitor 10 in a single housing, as is generally the case for atelevision.

Components of this control unit 6, necessary for the invention, are forinstance a communication interface 7, which, in particular, can receivesignals and data from the communication interface of the pointing device1, or transmit signals and data to the pointing device 1. Furthercomponents are a control point visualisation unit 8 and aninterpretation unit 9, the function of which will be explained below.

The coordinates x′, y′ of the control point C within the command areaA_(CM), received by the communication interface 7, are initiallyforwarded to the control point visualisation unit 8. With the aid ofthis unit 8, a cursor—for example a small square, cross, arrow orsimilar, is controlled such that the user can see the point in thecommand area A_(CM) at which he is apparently aiming the pointing device1. The coordinates x′, y′ of the control point C are also forwarded toan interpreting unit 9. This interpreting unit 9 serves to interpret theposition of the control point C in order to determine which of thedisplayed options M1, M2, M3, M4, M5, M6 in the command area A_(CM) theuser is selecting. To this end, the interpreting unit 9 must know thestatus of the menu currently shown in the command area A_(CM). Then,using the coordinates x′, y′ of the control point C, it can determine atwhich of the options M1, M2, M3, M4, M5, M6 the user is currentlypointing. The interpreting unit 9 can also be responsible for thecurrent status of the graphical user interface in the command area.

The control point visualisation unit 8 and the interpreting unit 9 mightbe realised in the form of software modules of a programmable processorof the control unit 6.

Evidently, the control unit or the device to be controlled D₁ willfeature a number of further components necessary or desirable for theoperation of such a device, such as a power supply, various interfacesand components for controlling the various applications of the device,for instance a receiver for receiving television signals in the case ofa television, or a DVD carriage in the case of a DVD recorder, etc. Forthe sake of simplicity, these components are not shown in FIG. 3.However, it will be clear to a person skilled in the art whichcomponents are necessary for each type of device.

Once again, it will be stressed that it is not necessary to realise theimage processing unit 3 and the control point processing unit 4 withinthe pointing device 1. One or both of these units 3, 4 could equallywell be realised within the control unit 6. In this case, for example,the target area images I generated by the camera 2 will be directlytransmitted, for example via the transmitter module 5, to the controlunit 6. Equally, the image processing unit 3 and the control pointposition calculation unit 4 might also be realised together in aseparate entity.

FIGS. 4 and 5 illustrate a situation in which two devices, each with itsown command area A_(CM), A_(CM′) are located directly beside each other.In such a situation, problems could arise when the control areas A_(CT),A_(CT′) associated with the command areas A_(CM), A_(CM′) would overlap.Were the user to point at an overlap area, this might result inindecision as to which of the two devices the user wishes to control.

FIG. 4 shows one solution, which ensures that no overlap whatever arisesbetween the control areas A_(CT), A_(CT′) of the neighbouring commandareas A_(CM), A_(CM′). This can be done by making known to the devicesor to the pointing device the locations of the different A_(CM),A_(CM′), and/or having previously assigned certain regions to theindividual command areas A_(CM), A_(CM′) within which the control areasA_(CT), A_(CT′) are permitted to move. FIG. 4 shows that this can leadto the control area A_(CT), A_(CT′) of a command area A_(CM), A_(CM′) isnot symmetric about its corresponding command area A_(CM), A_(CM′).Ideally however, the control area A_(CT) should extend proportionally ineach direction about the corresponding command area A_(CM), since thisoffers a more intuitive interaction.

An overlapping of the control areas A_(CT) can also be avoided by havingthe different virtual control areas A_(CT), A_(CT′) “repel” each other,i.e., by taking a neighbouring control area A_(CT), A_(CT′) intoconsideration whenever a control area A_(CT), A_(CT′) is being enlarged.If a neighbouring control area would be touched by a control areaA_(CT), A_(CT′) being enlarged, the neighbouring control area willshrink as the control area A_(CT), A_(CT′) is enlarged, or a furtherenlargement of the control area A_(CT), A_(CT′) will be prohibited.

FIG. 5 shows a further alternative. Here, the control areas A_(CT),A_(CT′) are permitted to extend until they touch a certain region, aso-called activation region A_(AC), A_(AC′), of a neighbouring commandarea A_(CM), A_(CM′). Thus, only one control area A_(CT), A_(CT′) isever active. Activation is effected when the user aims the pointingdevice in the activation region A_(AC), A_(AC′) of the correspondingcommand area A_(CM), A_(CM′). This means that the user can switch asdesired between command area A_(CM) and command area A_(CM′), dependingon which device he currently wishes to control.

At this point it will be explained in more detail, together with FIG. 6,how the position x, y of the real target point T, being aimed at withthe pointing device 1, can be determined with the aid of templates ofthe command area and/or its surroundings.

The user will not always aim the pointing device 1 at the command areaA_(CM) from directly in front—it is more likely that the pointing device1 will be aimed at a more or less oblique angle to the command areaA_(CM), since it is often more convenient to aim the pointing device 1than it is to change one's own position. This is illustrated in FIG. 6,which shows a schematic representation of a target area image Igenerated by a pointing device 1, aimed at a device D₂, in this case atelevision screen or computer monitor, from a distance and at an obliqueangle, so that the scale and perspective of the device D₂ in the targetarea A_(I) appear distorted in the target area image I.

A number of options M₁, M₂, M₃ can be seen on the display of the deviceD₂. The user, not shown in the diagram, may wish to select one of theseoptions M₁, M₂, M₃ with the aid of the pointing device 1. Also visiblein the display of the device D₂ is a cursor at the position of themomentary control point C, thus assisting the user who is aiming thepointing device 1. As explained above the current position of thecontrol point C is calculated depending on the position of the realtarget point T in a virtual extended control area A_(CT).

To find the position of the real target point T, the target area image Iis examined in more detail. Regardless of the angle of the pointingdevice 1 with respect to the device D₂, the target area image I isalways centred around the target point T. Therefore, this point isrelatively easily determined within the target area image I. It onlyremains to determine the relationship between the coordinates of thispoint T to the command area A_(CM) or the centre of the command areaA_(CM). Alternatively of course, the coordinates can be determinedrelative to another point in the command area A_(CM), for example to acorner point of a displayed user option, or to any notable point of themonitor such as a corner point, whereby these coordinates can then beconverted to the desired origin. Since the measurements of the virtualextended control area are preferably carried out in the same coordinatesystem, it is therefore trivial to determine the coordinates of theactual target point T within the virtual extended control area A_(CT).

For determination of the coordinates of the real target point T in thechosen coordinate system, the image processing unit 3 compares thetarget area image I with pre-defined templates TD₂.

Therefore, computer vision algorithms using edge- and corner detectionmethods are applied to locate points [(x_(a)′, y_(a)′), (x_(b)′,y_(b)′), (x_(c)′, y_(c)′)] in the target area image I which correspondto points [(x_(a), y_(a)), (x_(b), y_(b)), (x_(c), y_(c))] in thetemplate TD₂ of the device D₁.

Each point can be expressed as a vector e.g. the point (x_(a), y_(a))can be expressed as {right arrow over (v)}_(a). As a next step, atransformation function T is developed to map the target area image I tothe template TD₂:

${f(\lambda)} = {\sum\limits_{i}{{{T_{\lambda}\left( {\overset{->}{v}}_{i} \right)} - {\overset{->}{v}}_{i}^{\prime}}}^{2}}$

Where the vector {right arrow over (v)}_(i) represents the coordinatepair (x_(i), y_(i)) in the template TD₂, and the vector {right arrowover (v)}_(i)′ represents the corresponding coordinate pair (x′_(i),y′_(i)) in the target area image I. The parameter set λ, comprisingparameters for rotation and translation of the image yielding the mostcost-effective solution to the function, can be applied to determine theposition and orientation of the pointing device 1 with respect to thecommand area A_(CM). The distance between, for example, the pointingdevice 1 and the command area A_(CM) can be determined in this way.Furthermore—since the target point T is always the centre of the targetarea image I—the coordinates of the target point T in the template TD₂of the command area A_(CM) can be determined.

Although the present invention has been disclosed in the form ofpreferred embodiments and variations thereon, it will be understood thatnumerous additional modifications and variations could be made theretowithout departing from the scope of the invention. The pointing devicecan serve as the universal user interface device in the home or anyother environment with electrically or electronically controllabledevices. In short, it can be beneficial wherever the user can express anintention by pointing. Its small form factor and its convenient andintuitive pointing modality can elevate such a simple pointing device toa powerful universal remote control. As an alternative to the pen shape,the pointing device could for example also be a personal digitalassistant (PDA) with a built-in camera, or a mobile phone with abuilt-in camera. For the sake of clarity, it is also to be understoodthat the use of “a” or “an” throughout this application does not excludea plurality, and “comprising” does not exclude other steps or elements.A “unit” may comprise a number of blocks or devices, unless explicitlydescribed as a single entity.

1. A system for controlling a position of a control point in a commandarea, the system comprising: a graphical user interface including thecommand area indicating a user option; a pointing device comprising acamera for aiming in a direction of the command area to generate animage of a virtual control area; an image processing unit for processingthe image for determining the position of the target point at which thepointing device is aimed depending on the image and for calculating theposition of a control point depending on the position of the targetpoint; and wherein the dimensions and/or location of the virtual controlarea are defined according to momentary pointing conditions.
 2. Thesystem of claim 1 wherein the virtual control area at least partiallyoverlaps the command area.
 3. The system of claim 1, wherein thedimensions of the control area are defined according to a distancebetween the pointing device and the command area.
 4. The system of claim1, wherein the dimensions of the control area are extended if a distancebetween the pointing device and the command area exceeds a predeterminedlimit.
 5. The system of claim 1, wherein an extension factor, whichdefines the ratio of the dimensions of the control area to thedimensions of the command area, is proportional to a distance betweenthe pointing device and the command area.
 6. The system of claim 1,wherein the dimensions of the control area are extended if the pointingdevice is aimed at a target point outside of the current control area.7. The system of claim 1, wherein the dimensions of the control area aredecreased after a predetermined time period during which the pointingdevice is not aimed at a target point outside of a certain region of thecurrent control area.
 8. The system of claim 1, wherein the controlareas of adjacent command areas are chosen so that the control areas donot overlap.
 9. The method according to claim 1, wherein the commandarea can be activated by aiming at a certain region of the command area,and an extended control area associated with the currently activecommand area is chosen not overlap with at least a certain region of anadjacent command area.
 10. A method of controlling a position of acontrol point in a command area, the method comprising: providing agraphical user interface including the command area indicating a useroption; providing a pointing device comprising a camera for aiming in adirection of the command area to generate an image of a virtual controlarea; providing an image processing unit for determining the position ofthe target point at which the pointing device is aimed depending on theimage of the virtual control area, and for calculating the position ofthe control point depending on the position of the target point; andwherein the dimensions and/or location of the virtual control area aredefined according to momentary pointing conditions.